Data Integration Framework Research Articles

Advanced Framework for Multi-Modal Healthcare Data Integration: Leveraging HPC with GPU Computing and CNN Architecture in CDSS

In this study, we shall be looking at the challenges involved in integrating multi-modal healthcare data in the clinical decision support systems (CDSS). We propose the Automated Multi-Modal Data Integration (AMMI-CDSS) algorithm, which will utilize the latest high-performance computing (HPC) techniques such as the Convolutional Neural Network (CNN) architecture and the Graphics Processing Unit (GPU) computing to provide precise and rapid analysis. Which features will be extracted, multi-modal data will be merged, data will be prepared and algorithms developed in a distributed computing environment. We illustrate how AMMI-CDSS through the use of real world datasets such as wearable sensors data, medical imaging, genetic data, and electronic health records (EHRs), can improve the clinical decision support. By performing harmonization of the diverse data sources into a unique dataset after thorough data preprocessing and complex calculations, AMMI-CDSS provides the analysis with better quality and coherence.
 Our study allow us to make conclusion about how HPC-based CDSS models can be compared to conventional machine learning ones using their scalability and performance as key metrics. We enrich CDSS with the methodical framework for one-by-one testing and evaluation of proposed models and multi-modal healthcare data analysis. Future research might explore novel methods for integrating diverse types of healthcare data, as well as enhancing the HPC-based CDSS models by keeping them up-to-date.

Bayesian framework for multi-source data integration-Application to human extrapolation from preclinical studies.

In preclinical investigations, for example, in in vitro, in vivo, and in silico studies, the pharmacokinetic, pharmacodynamic, and toxicological characteristics of a drug are evaluated before advancing to first-in-man trial. Usually, each study is analyzed independently and the human dose range does not leverage the knowledge gained from all studies. Taking into account all preclinical data through inferential procedures can be particularly interesting in obtaining a more precise and reliable starting dose and dose range. Our objective is to propose a Bayesian framework for multi-source data integration, customizable, and tailored to the specific research question. We focused on preclinical results extrapolated to humans, which allowed us to predict the quantities of interest (e.g. maximum tolerated dose, etc.) in humans. We build an approach, divided into four steps, based on a sequential parameter estimation for each study, extrapolation to human, commensurability checking between posterior distributions and final information merging to increase the precision of estimation. The new framework is evaluated via an extensive simulation study, based on a real-life example in oncology. Our approach allows us to better use all the information compared to a standard framework, reducing uncertainty in the predictions and potentially leading to a more efficient dose selection.

Developing a goal-driven data integration framework for effective data analytics

Data integration plays a crucial role in business intelligence, aiding decision-makers by consolidating data from heterogeneous sources to provide deep insights into business operations and performance. In the big data era, automated data integration solutions need to process high volumes of disparate data robustly and seamlessly for various analytical needs or operational actions. Existing data integration solutions exhibit limited capabilities for capturing and modeling users' needs to execute on-demand data integration. This study, underpinned by affordance theory and the goal definition principles from the Goal-Question-Metric approach, designs and instantiates a goal-driven data integration framework for data analytics. The proposed innovative design automates data integration for non-technical data users. Specifically, it demonstrates how to elicit and ontologize users' data-analytic goals and addresses semantic heterogeneity, thereby recognizing goal-relevant datasets. In a structured evaluation using the context of counter-terrorism analytics, our design artifact shows promising performance in capturing diverse and dynamic user goals for data analytics and in generating integrated data tailored to these goals. Our research establishes a theoretical framework to guide future scholars and practitioners in building smart, goal-driven data integration.

Enhancing Online Food Delivery Systems through Comprehensive Text Analytics and Strategic Data Integration

Addressing challenges in the online food delivery system involves employing various data analytics techniques. Text Analytics, encompassing web analytics, social media analytics, stream analytics, and geospatial analytics, plays a pivotal role in managing and extracting valuable insights. The use of third-party systems by many companies to meet the demand for online food delivery presents issues related to control. Furthermore, information overload and poorly organized data contribute to observed problems. This research proposes effective data integration as a solution, facilitating strategic analytics for optimal system performance. Proper data sorting enables adaptive planning and priority shifts tailored to customer satisfaction. The framework of data integration is crucial in illustrating the comprehensive analysis of online food delivery systems. The report also delves into the challenges associated with implementing text analytics.

ScDREAMER for atlas-level integration of single-cell datasets using deep generative model paired with adversarial classifier

Integration of heterogeneous single-cell sequencing datasets generated across multiple tissue locations, time, and conditions is essential for a comprehensive understanding of the cellular states and expression programs underlying complex biological systems. Here, we present scDREAMER (https://github.com/Zafar-Lab/scDREAMER), a data-integration framework that employs deep generative models and adversarial training for both unsupervised and supervised (scDREAMER-Sup) integration of multiple batches. Using six real benchmarking datasets, we demonstrate that scDREAMER can overcome critical challenges including skewed cell type distribution among batches, nested batch-effects, large number of batches and conservation of development trajectory across batches. Our experiments also show that scDREAMER and scDREAMER-Sup outperform state-of-the-art unsupervised and supervised integration methods respectively in batch-correction and conservation of biological variation. Using a 1 million cells dataset, we demonstrate that scDREAMER is scalable and can perform atlas-level cross-species (e.g., human and mouse) integration while being faster than other deep-learning-based methods.

A Heritage Digital Twin ontology-based description of Giovanni Baronzio's “Crucifixion of Christ” analytical investigation

The methodologies of Cultural Heritage research and in particular those related to conservation and the analytical study of heritage assets and works of art, recently grouped under the term Heritage Sciences, engage a broad range of disciplines, each with its own characteristic workflows for generating data. Consequently, an emerging challenge is the need to define a digital framework for multi-source data integration, associated with a single heritage asset, but generated by various tools and methods which are often pursued by different research groups and at different times. This digital framework is discussed in this essay as the digital twin of a heritage asset, comprising of the documentation data associated with a heritage asset and its virtual representation. To best describe and define a Heritage Digital Twin ontology and its associated knowledge graph, we use a specific example drawing from art historical and analytical investigation of a 13th century Italian painting masterpiece, the Crucifixion of Christ, by Giovanni Baronzio, one of the leaders of the so-called Rimini School of painting which was greatly influenced by the work of the famous Renaissance master Giotto di Bondone.

A Digital One Health framework to integrate data for public health decision-making

The current implementation of One Health (OH) primarily focuses on multi-sectoral collaboration but often overlooks opportunities to integrate contextual and pathogen-related data into a unified data resource. This lack of integration hampers effective, data-driven decision-making in OH activities. In this perspective, we examine the existing strategies for data sharing and identify gaps and barriers to integration. To overcome these challenges, we propose the Digital OH (DOH) framework for data integration, which consolidates data-sharing principles within five pillars for the OH community of practice: (a) Harmonization of standards to establish trust, (b) Automation of data capture to enhance quality and efficiency, (c) Integration of data at point of capture to limit bureaucracy, (d) Onboard data analysis to articulate utility, and (e) Archiving and governance to safeguard the OH data resource. We discuss an upcoming pilot program as a use case focusing on antimicrobial resistance surveillance to illustrate the application of this framework. Our ambition is to leverage technology to create data as a shared resource using DOH not only to overcome current structural barriers but also to address prevailing ethical and legal concerns. By doing so, we can enhance the efficiency and effectiveness of decision-making processes in the OH community of practice, at a national, regional, and international level.

Data-driven integration framework for four-dimensional building information modeling simulation in modular construction: a case study approach

Abstract Modular construction is becoming more popular because of its efficiency, cost-saving, and environmental benefits, but its successful implementation necessitates detailed planning, scheduling, and coordination. Building information modeling (BIM) and four-dimensional (4D) simulation techniques have emerged as invaluable tools for visualizing and analyzing the construction process in order to meet these requirements. However, integrating distinctive data sources and developing comprehensive 4D BIM simulations tailored to modular construction projects present significant challenges. Case studies are used in this paper to define precise data needs and to design a robust data integration framework for improving 4D BIM simulations in modular construction. The validation of the framework in a real-world project demonstrates its efficacy in integrating data, promoting cooperation, detecting risks, and supporting informed decision-making, ultimately enhancing modular building results through more realistic simulations. By solving data integration difficulties, this research provides useful insights for industry practitioners and researchers, enabling informed decision-making and optimization of modular building projects.

A semi-automated hybrid schema matching framework for vegetation data integration

Integrating disparate and distributed vegetation data is critical for consistent and informed national policy development and management. Australia’s National Vegetation Information System (NVIS) under the Department of Climate Change, Energy, the Environment and Water (DCCEEW) is the only nationally consistent vegetation database and hierarchical typology of vegetation types in different locations. Currently, this database employs manual approaches for integrating disparate state and territory datasets which is labour intensive and can be prone to human errors. To cope with the ever-increasing need for up to date vegetation data derived from heterogeneous data sources, a Semi-Automated Hybrid Matcher (SAHM) is proposed in this paper. SAHM utilises both schema level and instance level matching following a two-tier matching framework. A key novel technique in SAHM called Multivariate Statistical Matching is proposed for automated schema scoring which takes advantage of domain knowledge and correlations between attributes to enhance the matching. To verify the effectiveness of the proposed framework, the performance of the individual as well as combined components of SAHM have been evaluated. The empirical evaluation shows the effectiveness of the proposed framework which outperforms existing state of the art methods like Cupid, Coma, Similarity Flooding, Jaccard Leven Matcher, Distribution Based Matcher, and EmbDI. In particular, SAHM achieves between 88% and 100% accuracy with significantly better F1 scores in comparison with state-of-the-art techniques. SAHM is also shown to be several orders of magnitude more efficient than existing techniques.

An Ontology-Based Framework for Geospatial Integration and Querying of Raster Data Cube Using Virtual Knowledge Graphs

The integration of the raster data cube alongside another form of geospatial data (e.g., vector data) raises considerable challenges when it comes to managing and representing it using knowledge graphs. Such integration can play an invaluable role in handling the heterogeneity of geospatial data and linking the raster data cube to semantic technology standards. Many recent approaches have been attempted to address this issue, but they often lack robust formal elaboration or solely concentrate on integrating raster data cubes without considering the inclusion of semantic spatial entities along with their spatial relationships. This may constitute a major shortcoming when it comes to performing advanced geospatial queries and semantically enriching geospatial models. In this paper, we propose a framework that can enable such semantic integration and advanced querying of raster data cubes based on the virtual knowledge graph (VKG) paradigm. This framework defines a semantic representation model for raster data cubes that extends the GeoSPARQL ontology. With such a model, we can combine the semantics of raster data cubes with features-based models that involve geometries as well as spatial and topological relationships. This could allow us to formulate spatiotemporal queries using SPARQL in a natural way by using ontological concepts at an appropriate level of abstraction. We propose an implementation of the proposed framework based on a VKG system architecture. In addition, we perform an experimental evaluation to compare our framework with other existing systems in terms of performance and scalability. Finally, we show the potential and the limitations of our implementation and we discuss several possible future works.

Multi-source data integration and multi-scale modeling framework for progressive prediction of complex geological interfaces in tunneling

A reliable geological model plays a fundamental role in the efficiency and safety of mountain tunnel construction. However, regional models based on limited survey data represent macroscopic geological environments but not detailed internal geological characteristics, especially at tunnel portals with complex geological conditions. This paper presents a comprehensive methodological framework for refined modeling of the tunnel surrounding rock and subsequent mechanics analysis, with a particular focus on natural space distortion of hard-soft rock interfaces at tunnel portals. The progressive prediction of geological structures is developed considering multi-source data derived from the tunnel survey and excavation stages. To improve the accuracy of the models, a novel modeling method is proposed to integrate multi-source and multi-scale data based on data extraction and potential field interpolation. Finally, a regional-scale model and an engineering-scale model are built, providing a clear insight into geological phenomena and supporting numerical calculation. In addition, the proposed framework is applied to a case study, the Long-tou mountain tunnel project in Guangzhou, China, where the dominant rock type is granite. The results show that the data integration and modeling methods effectively improve model structure refinement. The improved model's calculation deviation is reduced by about 10% to 20% in the mechanical analysis. This study contributes to revealing the complex geological environment with singular interfaces and promoting the safety and performance of mountain tunneling.

Ideas and perspectives: Beyond model evaluation – combining experiments and models to advance terrestrial ecosystem science

Abstract. Ecosystem manipulative experiments are a powerful tool to understand terrestrial ecosystem responses to global change because they measure real responses in real ecosystems and yield insights into causal relationships. However, their scope is limited in space and time due to cost and labour intensity. This makes generalising results from such experiments difficult, which creates a conceptual gap between local-scale process understanding and global-scale future predictions. Recent efforts have seen results from such experiments used in combination with dynamic global vegetation models, most commonly to evaluate model predictions under global change drivers. However, there is much more potential in combining models and experiments. Here, we discuss the value and potential of a workflow for using ecosystem experiments together with process-based models to enhance the potential of both. We suggest that models can be used prior to the start of an experiment to generate hypotheses, identify data needs, and in general guide experimental design. Models, when adequately constrained with observations, can also predict variables which are difficult to measure frequently or at all, and together with the data they can provide a more complete picture of ecosystem states. Finally, models can be used to help generalise the experimental results in space and time, by providing a framework in which process understanding derived from site-level experiments can be incorporated. We also discuss the potential for using manipulative experiments together with models in formalised model–data integration frameworks for parameter estimation and model selection, a path made possible by the increasing number of ecosystem experiments and diverse observation streams. The ideas presented here can provide a roadmap to future experiment–model studies.

Osteosarcoma Explorer: A Data Commons With Clinical, Genomic, Protein, and Tissue Imaging Data for Osteosarcoma Research.

Osteosarcoma research advancement requires enhanced data integration across different modalities and sources. Current osteosarcoma research, encompassing clinical, genomic, protein, and tissue imaging data, is hindered by the siloed landscape of data generation and storage. Clinical, molecular profiling, and tissue imaging data for 573 patients with pediatric osteosarcoma were collected from four public and institutional sources. A common data model incorporating standardized terminology was created to facilitate the transformation, integration, and load of source data into a relational database. On the basis of this database, a data commons accompanied by a user-friendly web portal was developed, enabling various data exploration and analytics functions. The Osteosarcoma Explorer (OSE) was released to the public in 2021. Leveraging a comprehensive and harmonized data set on the backend, the OSE offers a wide range of functions, including Cohort Discovery, Patient Dashboard, Image Visualization, and Online Analysis. Since its initial release, the OSE has experienced an increasing utilization by the osteosarcoma research community and provided solid, continuous user support. To our knowledge, the OSE is the largest (N = 573) and most comprehensive research data commons for pediatric osteosarcoma, a rare disease. This project demonstrates an effective framework for data integration and data commons development that can be readily applied to other projects sharing similar goals. The OSE offers an online exploration and analysis platform for integrated clinical, molecular profiling, and tissue imaging data of osteosarcoma. Its underlying data model, database, and web framework support continuous expansion onto new data modalities and sources.

An ontology-based methodology to establish city information model of digital twin city by merging BIM, GIS and IoT

With the development of digital city and smart city construction, the City Information Model (CIM) has played a critical role as a container of spatial–temporal data to establish the Digital Twin City. For a digital twin city, a virtual high-fidelity CIM model that corresponds closely to the real physical world is the premise and cornerstone of its construction. Therefore, the integration of BIM, GIS and IoT has become the preferred topic for researchers and has received much more attention from a wide academic circle. However, traditional integration mainly focuses on the conversion of both IFC and CityGML, and IoT data are also often used as visualizations. More importantly, the underlying data formats of GIS, BIM and IoT are still independent of each other without a unified data structure expression, so real data-driven analysis and decision-making cannot be implemented. This study aims to establish a general CIM ontology to integrate heterogeneous BIM, GIS and IoT data. First, the related work of BIM, GIS and IoT integration is studied and analyzed. A comparison of three mainstream approaches, data conversion, standard extension and data linking, is conducted, and it illustrates the advantages of ontology techniques in solving data interoperability problems. Second, a technical framework of BIM, GIS and IoT data integration based on ontology technology is proposed. The approach is mainly divided into five steps: geometry processing, data instantiation, ontology construction, ontology mapping and querying application. On the basis of the CIM ontology, an application ontology is built for a specific application domain to illustrate rule-based mapping, querying and inferring. Finally, the case study shows that the Ontology-based methodology in this paper has contributed to establish a general pattern for CIM data integration by mapping and linking concepts from the semantic level. It avoids changes in the original data sources and the missing data problem.

HIPPYlib-MUQ: A Bayesian Inference Software Framework for Integration of Data with Complex Predictive Models under Uncertainty

Bayesian inference provides a systematic framework for integration of data with mathematical models to quantify the uncertainty in the solution of the inverse problem. However, the solution of Bayesian inverse problems governed by complex forward models described by partial differential equations (PDEs) remains prohibitive with black-box Markov chain Monte Carlo (MCMC) methods. We present hIPPYlib-MUQ, an extensible and scalable software framework that contains implementations of state-of-the art algorithms aimed to overcome the challenges of high-dimensional, PDE-constrained Bayesian inverse problems. These algorithms accelerate MCMC sampling by exploiting the geometry and intrinsic low-dimensionality of parameter space via derivative information and low rank approximation. The software integrates two complementary open-source software packages, hIPPYlib and MUQ. hIPPYlib solves PDE-constrained inverse problems using automatically-generated adjoint-based derivatives, but it lacks full Bayesian capabilities. MUQ provides a spectrum of powerful Bayesian inversion models and algorithms, but expects forward models to come equipped with gradients and Hessians to permit large-scale solution. By combining these two complementary libraries, we created a robust, scalable, and efficient software framework that realizes the benefits of each and allows us to tackle complex large-scale Bayesian inverse problems across a broad spectrum of scientific and engineering disciplines. To illustrate the capabilities of hIPPYlib-MUQ, we present a comparison of a number of MCMC methods available in the integrated software on several high-dimensional Bayesian inverse problems. These include problems characterized by both linear and nonlinear PDEs, various noise models, and different parameter dimensions. The results demonstrate that large (∼ 50×) speedups over conventional black box and gradient-based MCMC algorithms can be obtained by exploiting Hessian information (from the log-posterior), underscoring the power of the integrated hIPPYlib-MUQ framework.

Flexible Data Refreshing Architecture for Health Information System Integration

Background: Having a consistent and unified view of heterogeneous distributed medical information sources is an inevitable need of health informatics. Integrating medical information of patients or about a disease, a treatment or side effects of a drug, etc, is very useful to help medical education, to achieve medical research goals and to provide the computer- based decision support systems. Contribution: This article proposes a flexible incremental update method for the materialized part of the integration system. It permits us to manage the integration system according to the characteristics of the data sources which can change. Method: This paper presents a hybrid data integration approach in which the materialized part of the system in mediator is the object indexation structure based on an instance classification of the sources objects which correspond to the global schema. The object identifier of each object in the indexation structure is materialized together with the attributes which are needed for the incremental updating of this indexation (classifying attributes). Results: The main idea of this paper is to develop a hybrid data integration framework, which represents a new aspect of a hybrid method focusing on flexible data refreshing. Conclusion: This hybrid approach implements a vertical hybrid approach. It means that at the mediator level, some data of each object are materialized and others are virtual.

Data Integration Framework to Collect Data from OT/IT Systems

Industry 4.0 and industrial data processing, due to its inherent possibilities, is gaining more and more emphasis in production companies these days. In a corporate environment, the age of equipment is extremely heterogeneous, in addition to state-of-the-art equipment, legacy systems can also be found in the machine park, which do not have appropriate communication protocols. Also, with the increase in the number of data sources, the management of data is becoming more and more challenging. Not only the operational technology, but also the connection of different IT systems and the extraction of data pose challenges. The different data processing use-cases using partly or entirely the same data sources, so it is necessary to extract and transmit the data to the target systems in a standard way, and avoiding an increase in the number of point-to-point interfaces. In this work we present a possible framework, to solve the above mentioned problems in industrial environment, with the introduction of standardized naming conventions, OT/IT gateways, data integration and distribution layers.

An integrated data management and informatics framework for continuous drug product manufacturing processes: A case study on two pilot plants

The pharmaceutical industry continuously looks for ways to improve its development and manufacturing efficiency. In recent years, such efforts have been driven by the transition from batch to continuous manufacturing and digitalization in process development. To facilitate this transition, integrated data management and informatics tools need to be developed and implemented within the framework of Industry 4.0 technology. In this regard, the work aims to guide the data integration development of continuous pharmaceutical manufacturing processes under the Industry 4.0 framework, improving digital maturity and enabling the development of digital twins. This paper demonstrates two instances where a data integration framework has been successfully employed in academic continuous pharmaceutical manufacturing pilot plants. Details of the integration structure and information flows are comprehensively showcased. Approaches to mitigate concerns in incorporating complex data streams, including integrating multiple process analytical technology tools and legacy equipment, connecting cloud data and simulation models, and safeguarding cyber-physical security, are discussed. Critical challenges and opportunities for practical considerations are highlighted.

ChroKit: a Shiny-based framework for interactive analysis, visualization and integration of genomic data.

We developed ChroKit (the Chromatin toolKit), an interactive web-based framework written in R that enables intuitive exploration, multidimensional analyses, and visualization of genomic data from ChIP-Seq, DNAse-Seq or any other NGS experiment that reports the enrichment of aligned reads over genomic regions. This program takes preprocessed NGS data and performs operations on genomic regions of interest, including resetting their boundaries, their annotation based on proximity to genomic features, the association to gene ontologies, and signal enrichment calculations. Genomic regions can be further refined or subsetted by user-defined logical operations and unsupervised classification algorithms. ChroKit generates a full range of plots that are easily manipulated by point and click operations, thus allowing 'on the fly' re-analysis and fast exploration of the data. Working sessions can be exported for reproducibility, accountability, and easy sharing within the bioinformatics community. ChroKit is multiplatform and can be deployed on a server to enhance computational speed and provide simultaneous access by multiple users. ChroKit is a fast and intuitive genomic analysis tool suited for a wide range of users due to its architecture and its user-friendly graphical interface. ChroKit source code is available at https://github.com/ocroci/ChroKit and the Docker image at https://hub.docker.com/r/ocroci/chrokit.

Meta-omics profiling of the gut-lung axis illuminates metabolic networks and host-microbial interactions associated with elevated lung elastance in a murine model of obese allergic asthma.

Obesity and associated changes to the gut microbiome worsen airway inflammation and hyperresponsiveness in asthma. Obesogenic host-microbial metabolomes have altered production of metabolites that may influence lung function and inflammatory responses in asthma. To understand the interplay of the gut microbiome, metabolism, and host inflammation in obesity-associated asthma, we used a multi-omics approach to profile the gut-lung axis in the setting of allergic airway disease and diet-induced obesity. We evaluated an immunomodulator, nitro-oleic acid (NO2-OA), as a host- and microbial-targeted treatment intervention for obesity-associated allergic asthma. Allergic airway disease was induced using house dust mite and cholera toxin adjuvant in C57BL6/J mice with diet-induced obesity to model obesity-associated asthma. Lung function was measured by flexiVent following a week of NO2-OA treatment and allergen challenge. 16S rRNA gene (from DNA, taxa presence) and 16S rRNA (from RNA, taxa activity) sequencing, metabolomics, and host gene expression were paired with a Treatment-Measured-Response model as a data integration framework for identifying latent/hidden relationships with linear regression among variables identified from high-dimensional meta-omics datasets. Targeting both the host and gut microbiota, NO2-OA attenuated airway inflammation, improved lung elastance, and modified the gut microbiome. Meta-omics data integration and modeling determined that gut-associated inflammation, metabolites, and functionally active gut microbiota were linked to lung function outcomes. Using Treatment-Measured-Response modeling and meta-omics profiling of the gut-lung axis, we uncovered a previously hidden network of interactions between gut levels of amino acid metabolites involved in elastin and collagen synthesis, gut microbiota, NO2-OA, and lung elastance. Further targeted metabolomics analyses revealed that obese mice with allergic airway disease had higher levels of proline and hydroxyproline in the lungs. NO2-OA treatment reduced proline biosynthesis by downregulation of pyrroline-5-carboxylate reductase 1 (PYCR1) expression. These findings are relevant to human disease: adults with mild-moderate asthma and BMI ≥ 25 had higher plasma hydroxyproline levels. Our results suggest that changes to structural proteins in the lung airways and parenchyma may contribute to heightened lung elastance and serve as a potential therapeutic target for obese allergic asthma.